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The strong correlation effects on topological insulator are studied in a two-sublattice system with 
an onsite single-particle energy difference A between two sublattices. At A = 0, increasing the 
onsite interaction strength U drives the transition from the quantum spin Hall insulating state 
to the non-topological antiferromagnetic Mott-insulating (AFMI) state. When A is larger than a 
certain value, a topologically trivial band insulator or AFMI at small values of U may change into a 
quantum anomalous Hall state with antiferromagnetic ordering at intermediate values of U. Further 
increasing U drives the system back into the topologically trivial state of AFMI. The corresponding 
phenomena is observable in the solid state and cold atom systems. We also propose a scheme to 
realize and detect these effects in cold atom systems. 
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The recent prediction and experimental observation of 
the quantum spin Hall (QSH) state raise great interests 
in the study of topological phase transitions [lHl] ■ A sim- 
ple model for quantum spin Hall effect (QSHE) can be 
constructed by coupling theparticles to a spin-dependent 
effective magnetic field 0, 0| . By extending such model 
to the many-body case, the possible appearance of frac- 
tional QSH states was studied d, [si— TlQIj - However, the 
more realistic situation for the QSH effect is the topo- 
logical insulator (TI) which can be achieved with the in- 
verted band structure in the presence of strong spin-orbit 
coupling. In the single particle picture the TI is charac- 
terized by the nontrivial Zi topological invariant, with 
one-dimensional gapless helical modes existing along the 
edge for two dimensional (2D) materials [l|, and (2+1)- 
dimensional ((2+l)D) gapless Dirac modes on the surface 
for the three dimensional (3D) case A natural gen- 
eralization of the TI is to consider the many-body effect, 
of which the simplest way is to combine the TI and the 
Fermi-Hubbard model by considering the on-site interac- 
tion. Different correlation effects on the TI are supposed 
to be obtained by adjusting the interaction strength. 

In this work we study the correlation effect on TI in 
a quasi 2D square lattice in the presence of a staggered 
sublattice potential A, which is shown to have a pro- 
nounced effect on the topological properties of the an- 
tiferromagnetic Mott insulator (AFMI) obtained in the 
large on-site Hubbard interaction U . While in the case 
A < A c , the AFMI is always topologically trivial, when 
A > A c it can be topologically non-trivial and exhibits 
the quantum anomalous Hall effect. 

We start with the quasi 2D square lattice model at half- 
filling depicted in Fig. QJa-b), with the on-site Hubbard 
interaction U and a single-particle sublattice energy dif- 
ference A between A and B sites. Similar to Haldane's 
original idea for QAHE [ll| and the Kane-Mele (KM) 
model for QSHE [l|] , we introduce a spin-dependent stag- 
gered magnetic flux in the lattice, which leads to a Peierls 
phase <j) for the nearest-neighbor-site hopping along the 




FIG. 1. (Color online) Configuration of 2D anisotropic square 
lattice for spin-up (a) and spin-down fermions (b); (c-e) Single 
particle bulk spectrum; (f) Single particle edge spectrum. 

marked direction (Fig. l(a-b)). The Hamiltonian in the 
tight-binding form reads H = Hq + Hi nt , with 

H = - t (cos<f) + is z i>ij sin 4>)& a}i Cb,j - 

<i,j> 

~ t^U^-j + A^(^, a -rn, b ), 

1=1,2 

H m t = Y2 Un t fn % i, (1) 

i 

where = (cy^, c^), My = 1 (-1) for hopping along (op- 
posite to) the marked direction, to is the nearest-neighbor 
hopping coefficient and U the intrasite interacting energy. 
In this model we consider the local orbitals on A and B 
sites to be anisotropic and thus the next-neighbor hop- 
ping (in e\ 2 directions) coefficients t a i ^ tu with I = 1, 2. 
Although the above system serves as a toy model, we 
stress later that the physics studied in the following is 
quite general. Furthermore, we shall propose a scheme 
to observe the phenomenon with cold atoms. 
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Before moving to the study of the correlation ef- 
fect, we give a quick description of the topological 
phase transition in the absence of interaction. For con- 
venience we transfer the original Hamiltonian into k 
space H = £ k #(k)«o(k)d(k) + H int with C(k) = 
(c Q f(k), 4t(k)j c o4 .(k), c H (k)) T and obtain 



H (k) = ^]d Q (k)r Q , 



2[/ 



ff-tni — ^,k'+qt^,k-qi^ I k+qJ.C Al .k'-qt; (2) 



kk'q 

fi—a,b 



where d\ = — to cos <f>(cos k x a + cosk y a),d2 = A — 
2(i i + t a2 - hi - t b2 ) cos k x a cos k y a + 2(t al + t b2 - t a2 - 
t bl ) sin k x a sin k y a, d$ = = 0, d 5 = — 2< sin 0(cos fc^a— 
cos fc y a), T Q are defined via {r Q } = {<r x ®I,a z ® I, a y ® 
s Xl <jy ® Sy,a y <£> s 2 } and iVo is the number of particles. 
For convenience we choose in the following = 7r/4, t a i > 
tbi,t a2 — tb2 and denote by t\ — t a \ — t&i. We can then 
simplify the coefficient ^(k) = A + 2ti(sin k x a sin k y a — 
cos k x a cos fc y a). Noting that T Q are even under time- 
reversal (TR) transformation and d a (k) — d a (— k), thus 
the above Hamiltonian satisfies the TR symmetry. 

The single particle spectrum of Hq (k) is given in Fig. 
QJc-e). When t\ = and A > 0, the system is a trivial 
band insulator. When 2t\ — A, the system becomes a 
semi-metal with the gap closed at k x — — k y = ir/2 and 
—ir/2 for spin-up and -down states, respectively. Around 
gap closing points the system is described by the mass- 
less Dirac Hamiltonians. Furthermore, when 2ti > |A|, 
the gap opens again and the mass terms of the Dirac 
Hamiltonians on the two points change sign relative to 
those in the case 2t± < |A|, which leads to a quantum 
jump (±e 2 /h) of the Hall conductance for each Dirac 
Hamiltonian [12j. Therefore the charge Hall conductiv- 
ity (CHC) reads now trj = —o xy = e 2 /h for spin-up and 
-down states. For this we know the spin Chern number 
of the present system: C sp i n = ±2 when 2ti > |A| and 

a 



spin 



otherwise, while the total CHC is always zero 
due to TR symmetry in the single-particle picture. The 
gapless helical edge modes on the boundaries at x = 
and x = L are shown in Fig. QJf), and such modes are 
protected by a nontrivial Z 2 topological number. In solid 
state systems, the edge modes can be detected by mea- 
suring the effective one-dimensional (ID) channel con- 
ductance 0, and in cold atoms, these modes may be 
detected with light Bragg scattering [l3| . 

Now we proceed to study the correlation effect on the 
topological phase transition. When \U\ is much smaller 
than the single-particle bulk gap, i.e. \U\ <C 2t\ — |A|, 
only the edge states will be scattered by particle-particle 
interaction, with the bulk states unaffected. In this way 
the effective Hamiltonian for edge modes can be de- 
scribed by ID helical Luttinger liquid model with only 
the forward scattering, which cannot open a gap but leads 



to a renormalization of the group velocity of edge states 
according to the standard bosonization approach 14 1. 

It is more interesting to consider the strong interacting 
regime where U is larger than the bulk gap. In this case 
the bulk physics will be changed and can be studied with 
Hartree-Fock mean field approach. For the half-filling 
system, the AF order (spin density wave (SDW)) will 
appear when the interacting strength U exceeds some 
critical value. We can introduce the mean-field staggered 
AF order parameter with rrn = (n^) — {n^) — sgn(z)TO, 
where sgn(i) = ±1 for A and B sub-lattices, respectively. 
Then the mean-field interacting Hamiltonian reads 



U 



u 



]Tct(k)[2™r 15 -m?]C(k)(3) 



with m = riif + riii and T 15 = s z ®<r z . Note the inversion 
symmetry of the present system is broken when A ^ 0, 
for which the occupation numbers (n Q ,) (at A sites) and 
(ribi) (at B sites) are generally different. For convenience 
we denote by (n a i) = 1 + (Sn) and {nu) = 1— (Sn), where 
Sn characterizes the difference between the occupation 
numbers and should be determined self-consistently. By 
combination of Eqs. ((2|) and ((3]) we get now the Hamil- 
tonian H m = ^ k Ct(k)^ m) C(k) + ^C/(l + (5n) 2 + m 2 ), 

where ?^ m) (k) = H (k.) + d 15 T 15 with d X5 = mU/2. 
Thus the nonzero AF order m gives rise to a new term 
to the original single particle Hamiltonian and sponta- 
neously breaks the TR symmetry of the system. To cal- 
culate (Sn), we can diagonalize T-Lq (k), which gives four 
eigenstates \x a p(m,k)) = , fi"f, ^"j 3 , } T , with 
a, /3 — ±1 and the corresponding eigenvalues E a p = 
ay/d 2 + d\ + (d 2 + /?g?i 5 ) 2 . The expectation value of 
(Sn) is determined by 



(fc(m,A)) = ^£/(i^)(| M « 



a/3 1 2 
■tl 



a/3k 

Ki 2 - 



(4) 



where f(E a p) is the fermi distribution function. Bear in 
mind these results, we obtain the mean field free energy 



F HF {m) = E aP (m,k)f(E aP ) + -fu(l + m 2 ) 

a/3k 

+ ^U(Sn(m,A)) 2 . (5) 

Combining the eqs. (j4]) and ([5]) we can solve the min- 
imum point of the free energy as a function of AF or- 
der to at each value of U which increases from zero to 
some finite number. It is expected when U > U c \, the 
AFMI phase occurs with the value to > at the min- 
imum point of free energy. The key thing is that once 
we obtain the AFMI phase at U > U c \, we can fur- 
ther determine the topological properties of the system 
by analyzing the spectrum of the Hamiltonian T-i^ (k) , 
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in which the nonzero term dis = mU /2 results in a 
spin-dependent renormalization to original single parti- 
cle bulk gap. The bulk gaps for the spin-up and spin- 
down states read now £ ga pt = |2ti — |A — mU/2\\ and 
£ ga pi — \2t\ — A—mU /2|, and the topology of spin-up and 
spin-down branches will be determined separately. For 
spin-up branch, the state is topologically nontrivial when 
| A - mU/2\ < 2ti and trivial when |A - mU/2\ > 2*i, 
while for spin-down branch, the corresponding conditions 
become \A + mU/2\ < 2tx and \A + mU/2\ > 2*i, respec- 
tively. In the following numerical study we assume the 
parameters A, m > (other cases of negative parameters 
can be discussed similarly) , in which situation one can see 
the topology of the spin-down states is more fragile to the 
interaction than that of the spin-up branch. Intuitively, 
this is because for spin-up states the interaction compen- 
sates the onsite energy difference in the mean field picture 
while for spin-down states it enhances such difference and 
blocks the nearest-neighbor hopping. 



md 1 









H 












\ 


















h = 0,2*0 


0.5 1 


(a) 


2 2 

A/to 




A,2ti Ai 



A/i 



FIG. 2. (Color online) (a) The mean field value {5n(m, A)) 
(inset) and the value of AF order m c i at the transition point 
U = Uci; (b) Relation between U c \,U C 2Uc3 and A, where 
A c ~ 0.26£o and Ai ~ 0.76to- The results are obtained at 
zero temperature. 

The mean field value (Sn(m, A)) is given in the inset of 
Fig. [21 with which one can further determine the spon- 
taneous AF order m c \ at the AF phase transition point 
U = U c i, as shown in Fig. [2] (a). For small A (< 2ti) 
where the system is in QSH phase in the non-interacting 
regime, the value of m c \ is finite, and this indicates the 
phase transition is of the first order. As A increases the 
first order transition is softened and becomes the second 
order one around A = 2t\, at which point m c \ varnishes. 
Further increasing A the phase transition returns to the 
first order. Another interesting property is the depen- 
dence of the critical interaction U c \ on A (Fig. [2](b), blue 
line). In the case A < 2t±, we find U c \ is a monotonously 
decreasing function of A and changes rapidly at A = 2t\, 
while for A > 2t\ it becomes a monotonously increasing 
function of A. This is because the original single-particle 
bulk gap decreases to zero when A increases to be 2t\, 
and further increasing A enlarges the single-particle bulk 
gap. These properties lead to important phenomena for 
the topological phase transition described below. 



First, when the on-site energy difference is smaller 
than some critical value A < A c (< 2ii), we find 
|A + m c iU c i/2\ > 2t x and |A - m c \U cl /2\ > 2t 1: 
which indicates once the AF transition occurs both spin 
branches become topologically trivial. In this case, no 
gapless edge mode exists at the AFMI phase. Sec- 
ond, when A c < A < 2t\, we have for the spin-down 
states | A + m c \\J c \j2\ > 2£i , while for spin- up states 
I A — to c iC/ c i/2| < 2t\. This means for the case U > U c \, 
the spin-down branch becomes trivial but the spin-up 
branch keeps topologically nontrivial for U c \ < U < U c j,. 
Here U C 3 is determined by m C 2,U c ?,/2 — A = 2t\ with m C 3 
obtained at U = U c ^. In this way, the quantum anoma- 
lous Hall (QAH) insulator which has a quantized CHC 
(e 2 /h) associated with the AF phase is obtained. Third, 
when 2ti < A < Ai with Ai depending on the lattice 
parameters such as hopping coefficients, the system is in 
the trivial band insulator phase for the weak interacting 
regime (U < U c i). However, it is quite interesting that 
in the region U c i < U < U C 3, we find again the phase 
that the QAH insulator with AF ordering. Finally, for 
A > Ai, increasing the interacting energy to U > U c \ 
first drives the system from trivial band insulator to the 
non-topological AFMI. However, further increasing U to 
the region £/ c2 < U < U C 3 again leads to the topological 
phase of QAH and AF insulator, with U C 2 determined 
via A — m C 2U C 2/2 = 2t\. For all these cases, when the 
interacting strength U > U C 3, the system turns to the 
topologically trivial AFMI phase. Fig. [3] gives the mean 
field phase diagram in different parameter regimes. 
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FIG. 3. (Color online) Mean field phase diagram with the 
parameter t\ = 0.2£o- 

The correlation effect obtained here is not restricted 
in square lattice. For the KM model in the honey- 
comb lattice, when both the Hubbard-type interaction 
and sublattice on-site energy difference are present, we 
have confirmed the same phase diagram given in Fig. [3] 
First-principle calculation shows that the physics in the 
transition metal oxide Na2lr03 [15| is captured by the 
KM-like model, and thus this material is a natural can- 
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didate to observe the above correlation physics. Alter- 
natively, we propose a novel scheme with cold atomic 
platform to study these phenomena. The square lat- 
tice can be realized by the periodic optical potential 
Viatt = -Vo(cos 2 k Q x + cos 2 h y) - Vi sin 2 [fc (£ + v)/% 
which is achievable with the experimental platform by 
Porto's group at NIST (Fig. g[a)) [16] with a minor 
modification. The second term of the potential (with 
depth V\) contributes to an anisotropic term, due to 
which the local s-orbital on the A sites extends longer 
than that of B sites in the e\ direction. This leads to 
the hopping coefficient t\ = t a \ — t^i > 0. Further- 
more, the on-site energy difference between the s-orbitals 
on the A and B sites is controllable and is obtained by 
A = Vi + 2E 1 r /2 [{V a - Vx/2) 1 / 2 - (V + Vi/2) 1 / 2 ], with 
E r = Y?k\j2m the recoil energy. 




repulsive by Feshbach resonance (l9j . 

In conclusion, we have studied the correlation effect 
on topological insulator in a square lattice. A rich phase 
diagram is obtained by varying the staggered sublatticc 
potential A and the Hubbard on-site interaction U. Es- 
pecially, for the case A > A c , we see the increasing U 
can drive the system into the topological antiferromag- 
nctic Mott insulating phase which exhibits the quantum 
anomalous Hall effect. The observation of these corre- 
lation physics in solid state and cold atom systems is 
discussed. Several interesting issues following the present 
study deserve future efforts in the research. For example, 
in square lattice, the frustration appears when the mag- 
nitudes of the hopping coefficients t\ ~ to- I n this case 
the spin liquid phase with gapless spinon excitations may 
be obtained in the intermediate interaction strength [20| . 
It is especially interesting to find out in this model the 
gapless spinon excitation is chiral or helical in different 
parameter regimes. 

X.J.L thanks Congjun Wu, J. Sinova, Ar. Abanov and 
Chia-Ren Hu for fruitful discussions and helpful com- 
ments. This work is supported by NSF under Grant No. 
DMR-0547875. 



FIG. 4. (Color online) (a) Cold atoms trapped in a square 
lattice based on the set-up in Porto's group at NIST pj]. (b) 
Generation of spin-dependent gauge potential with 6 Li atoms. 

The candidate for the fermi atoms can be 6 Li and 40 K, 
etc, and the hyperfine levels of 6 Li atoms are sketched 
in Fig. 0Jb). The periodic spin-dependent gauge poten- 
tial can be generated by coupling the atomic hyperfine 
levels to radiation. In Fig. 4(b) the a± transitions can 
be realized by a single laser with Rabi-frequency £l a — 
e + f2 e ifcl(:c+2 ' ) +e^fl e~ ikl( - x+y '> with e± representing the 
circular polarizations of photons, while the tt transitions 
are achieved by the laser with il^ — e^fli sin[fc (^ — 
y)/2 + tt/4]. For this the <r+ and the left hand tt transi- 
tions consist of a A system, while cr_ and the right hand 
tt transitions consist of another A' system. It is known 
each A system has a dark-state solution with which we 
define the (pseudo)spin states by = cos6>||,— ^) — 
sin(9e-^|§,i) and |^) = cos<9|±, \) - smfle^lf, -§) 
with 8 defined via tan 8 — Sli sin[fco(£ — y)/2 + 7r/4]/f2o 
and cp = k\(x + y). The spin-dependent gauge potential 
in the defined spin- 1/2 subspace is obtained straightfor- 
wardly by A(r) = TT ^^L_e lSz with 

7 = f2i/\/2^o, which gives rise to a spin-dependent peri- 
odic Peierls phase only for the nearest-neighbor-site hop- 
ping, as required in the present model. The interaction 
strength in cold atoms can be readily manipulated from 
weak or zero to strong regime, and from attractive to 
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